Skip to content

fix(fuse): isolate partition and cluster layouts - #20221

Open
KKould wants to merge 5 commits into
databendlabs:mainfrom
KKould:fix/fuse-partition-cluster-layout
Open

fix(fuse): isolate partition and cluster layouts#20221
KKould wants to merge 5 commits into
databendlabs:mainfrom
KKould:fix/fuse-partition-cluster-layout

Conversation

@KKould

@KKould KKould commented Jul 29, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

  • Remove the implicit semantic that prepended PARTITION BY expressions to CLUSTER BY keys
  • Persist partition statistics independently from cluster statistics
  • Keep compaction and reclustering selection, windows, and tasks within a single partition
  • Preserve input order in TransformPartitionBy and only split blocks at adjacent partition boundaries
  • Conservatively skip partition-aware reclustering when legacy metadata has no partition statistics

Tests

  • Unit Test
  • Logic Test
  • Benchmark Test
  • No Test - Explain why

Type of change

  • Bug Fix (non-breaking change which fixes an issue)
  • New Feature (non-breaking change which adds functionality)
  • Breaking Change (fix or feature that could cause existing functionality not to work as expected)
  • Documentation Update
  • Refactoring
  • Performance Improvement
  • Other (please describe):

This change is Reviewable

@github-actions github-actions Bot added the pr-bugfix this PR patches a bug in codebase label Jul 29, 2026
chatgpt-codex-connector[bot]
chatgpt-codex-connector Bot reviewed Jul 29, 2026
View reviewed changes

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d6ea3e0da0

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/query/storages/fuse/src/table_functions/clustering_information.rs
Comment thread src/query/storages/common/table_meta/src/meta/column_oriented_segment/segment_builder.rs
@KKould
KKould force-pushed the fix/fuse-partition-cluster-layout branch 2 times, most recently from 396bffa to d6ea3e0 Compare July 29, 2026 10:19
@KKould

KKould commented Jul 29, 2026

Copy link
Copy Markdown
Member Author

@codex review

@KKould KKould self-assigned this Jul 29, 2026
chatgpt-codex-connector[bot]
chatgpt-codex-connector Bot reviewed Jul 29, 2026
View reviewed changes

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d6ea3e0da0

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/query/storages/fuse/src/table_functions/clustering_information.rs
@KKould
KKould force-pushed the fix/fuse-partition-cluster-layout branch from d6ea3e0 to 50895ad Compare July 29, 2026 13:32
sundy-li
sundy-li requested changes Jul 29, 2026
View reviewed changes

@sundy-li sundy-li left a comment •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found two issues that need to be addressed:

  1. [P1] The full workspace test suite does not compile. After adding BlockMeta.partition_stats, the BlockMeta struct literal at src/query/storages/common/cache/src/manager.rs:1294 was not updated with the new field. The linux / check, linux / test_unit, and mac_check jobs all fail with E0063. I can also reproduce it locally with cargo test -p databend-storages-common-cache --lib --no-run.

  2. [P2] Clustering introspection can still reuse legacy physical cluster statistics for partitioned tables. Historical cluster_stats.min/max vectors contain (partition keys..., cluster keys...), while clustering_information and clustering_statistics now prepare expressions for only the logical cluster key and still pass the same cluster_key_id to get_min_max_stats. Since that function returns the cached vectors whenever the key ID matches without validating their dimensions, clustering information may be displayed or calculated using the partition prefix. Please recompute from col_stats when the cached statistics dimensions do not match the logical expression count, or otherwise explicitly detect the legacy layout.

The focused Fuse test suite passes (cargo test -p databend-common-storages-fuse --lib: 103 tests), but it does not cover these two issues.

@KKould
KKould force-pushed the fix/fuse-partition-cluster-layout branch from 50895ad to fe00398 Compare July 30, 2026 02:47
@KKould

KKould commented Jul 30, 2026

Copy link
Copy Markdown
Member Author
  1. [P2] Clustering introspection can still reuse legacy physical cluster statistics for partitioned tables. Historical cluster_stats.min/max vectors contain (partition keys..., cluster keys...), while clustering_information and clustering_statistics now prepare expressions for only the logical cluster key and still pass the same cluster_key_id to get_min_max_stats. Since that function returns the cached vectors whenever the key ID matches without validating their dimensions, clustering information may be displayed or calculated using the partition prefix. Please recompute from col_stats when the cached statistics dimensions do not match the logical expression count, or otherwise explicitly detect the legacy layout.

This feature has not been deployed in the actual production environment, so I do not believe compatibility is required.

@KKould
KKould force-pushed the fix/fuse-partition-cluster-layout branch from fe00398 to a805179 Compare July 30, 2026 04:19
@KKould
KKould force-pushed the fix/fuse-partition-cluster-layout branch from a805179 to 37802fd Compare July 30, 2026 06:39
@KKould
KKould requested a review from sundy-li July 30, 2026 16:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@chatgpt-codex-connector chatgpt-codex-connector[bot] chatgpt-codex-connector[bot] left review comments
@zhyass zhyass zhyass approved these changes
@zhang2014 zhang2014 Awaiting requested review from zhang2014
@b41sh b41sh Awaiting requested review from b41sh
@sundy-li sundy-li Awaiting requested review from sundy-li

Requested changes must be addressed to merge this pull request.

Assignees

@KKould KKould

Labels

pr-bugfix this PR patches a bug in codebase

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@KKould @sundy-li @zhyass